1. Field of the Invention
The present invention relates to the collection of video data in a moving vehicle from a plurality of video cameras with time correlated attitude and spacial information.
2. Description of the Prior Art
Previous apparatus has been devised to collect video data from a moving vehicle. In Lachinski et al., U.S. Pat. No. 5,633 946, incorporated herein by reference, analog video data is collected from a number of analog video cameras. Each camera is interrogated by a central processor in a preprogrammed sequence after which the camera resets to capture a video frame for transmission to a central processor for storage. The image response time from each camera is stored with the video image for later processing. Though a significant advance in the art, this approach can result in relatively large timing errors because the exact time of scan initiation is somewhat different for each camera because of the spatial distances and there is no attempt to account for different transmission times. This results in a different time offset of video images with respect to the response time making target identification difficult.
This problem may be further exacerbated if there is a need to move the cameras in relation to one another. This situation can result not only timing errors but varying timing errors.
An additional problem with the Lachinski et al. technique is that the video data is gathered and stored as video images on a video recorder. The actual processing of the images involves playback of the video recording. Therefore, the data must be gathered in real time and yet the final processing cannot be in real time.
The present invention overcomes the disadvantages of the prior art by providing a system in which the video images of a plurality of video cameras can be time tagged using a system wide time standard and digitized on a frame by frame basis at each station. As a result, the time stamped and digitized images can be processed at any time, including immediately at the data collection site, or at any convenient time thereafter. Furthermore, because each frame to be processed is time stamped using a system wide standard, the various individual images, including those for the same or different cameras, can be easily correlated in near real time or at anytime without the timing errors found in the prior art systems. Because the frames are digitally time stamped, the various individual cameras are free to operate asynchronously in the analog domain.
Though the present invention applies equally to a wide range in the number of individual video cameras, in the preferred mode, eight camera stations are used. Each station has an analog video camera and a video digitizer or digital camera which converts the video camera analog output to a digital image. A computer in each station time stamps the initiation of the capture of each frame in relation to a system standard, compresses the digital data, and stores the data in local memory storage.
A master processor, which controls the entire system. has a hard drive for longer term digital image storage. The master processor and the eight stations are interconnected by a Dual FDDI Token Ring local area network (LAN) having a control ring for the two way transmission of timing and control signals and a data ring for the two way transmission of image data. Each station provides its station identification, the digitized video data and a delta time, described below, indicating the time delay from the system time token to the time at which the current video image scan began, over the data ring to the master processor which stores the data from all of the stations on the hard drive.
The master processor requests data for the current video frame from the stations by broadcasting a token over the control ring. This token is received by all stations. Each station has a local clock, periodically synchronized with the system time standard, which is used to determine the delta or offset time from beginning scan time of the current video frame until the time the token was received. Thus, the relative timing amongst all of the individual frames is known. This value can be calculated by a ranging process that measures the flight time of the time stamp through the networked stations.
In the preferred mode because the current video frame is the one requested, this results in the delta time for the camera in any station being no greater than one video frame time. Since the scan time for a video frame is approximately {fraction (1/30)} of a second, the delta time is not greater than that amount. Thus, the delta or offset time is typically small, insuring a high degree of accuracy in correlating the frames from the various cameras, not withstanding a separate and asynchronous clock within each of the eight stations.
Unlike the prior art systems which rely upon synchronizing the video cameras, in accordance with the present invention, each camera may have a separate synch generator and therefore the delta time may be different for each camera. In essence, the transmission time of the token itself provides the timing reference for the cameras by means of the delta offset time. While there is also an inherent time delay for the transmission of the token from the master processor to each station, the time delay for each station is fixed and can be measured. A predetermined fixed transmission time for each station is simply added to the delta time for each station in later position calculations to more accurately determine the exact scan time initiation for the video frame.
Each station continues processing its current video image after receipt of the broadcast token. The first station to complete processing the entire current video image transmits a token over the control ring. Relative time correlation amongst the video frames from different cameras is reestablished because each contains a time stamp.
The station then transmits the compressed digital image data, the delta time and its station identifier to the master processor over the data ring. The master processor stores this data in its own memory storage. The next station to complete processing the current video image repeats the above procedure. This process continues until all eight of the stations have completed processing the entire current video image and provided the data to the master processor for storage. After all eight stations have responded, this completes the data transmission process for the master processor for a single master processor token transmission.
The essence of the present invention is that data provided is for the current images, and the beginning scan time for each current image can be determined accurately. This approach not only reduces the offset time between camera images, since the video scan time for one frame is relatively small, but also accurately determines the offset times between camera images. Further, since the time of transmission between the master processor and each station can be measured and included with the delta time computations, the actual acquisition time for the initiation of each video image can be determined with a high degree of accuracy. This results in position calculations based on this data being considerably more accurate than before.
In the preferred mode of the present invention, the vehicle has eight stations with each station having a single camera. The stations are arranged in pairs with the cameras in a pair both aimed in the same direction. While the camera pairs are both aimed in the same direction, each pair is aimed at right angles to all others, namely: forward, aft, perpendicular right and perpendicular left relative to the fore and aft line of the vehicle. One camera of each camera pair is focused for a zoomed-in close-up shot while the other camera is focused for a wide-angle shot. The zoomed camera shot provides greater detail but a smaller field of view than the wide angle shot. This may also be done with a single panoramic camera mounted at each station at 90% rotations to vertical and sampled more frequently.
In this arrangement each camera station responds to a request for data only upon completion of scanning the current image. This results in interleaving the data from the eight camera stations in a specific, preprogrammed order. However since the data from each station includes a station identifier, the images pertaining to each camera can be later identified by means of this station identification.
A global positioning system (GPS) mounted on the vehicle provides latitude, longitude and altitude, and their respective rates plus an accurate time signal. The position signals and their rates are received stored by the master processor along with all the digitized video data from each station. The time signal is also received, stored and correlated with data requests. In addition, the time signal is used to maintain the accuracy of a master processor time keeper and to periodically reset the time keeper in each station.
A fixed base GPS, located near the vehicle, is operated at the same time as the vehicle system, which includes the vehicle mounted GPS, to collect data which has the same intentional system errors introduced as does the vehicle mounted system. This permits correcting the intentional errors introduced into the GPS information by later differential processing of the fixed base GPS data versus data from the vehicle on-board GPS data.
The GPS system only updates the output data approximately three times per second, and occasionally there are no navigational satellites available to update the GPS data. An Inertial navigational System also mounted on the vehicle is provided to supplement the GPS data. The Inertial navigational system data is also received and stored by the master processor in its hard drive. Data from the Inertial navigational System includes not only position and altitude information and their rates, but also the yaw, pitch and roll information of the vehicle and their rates. All of this inertial data, updated approximately thirty times per second, is sent to the master processor where it is stored in memory along with a corresponding time signal.
The inertial information is used to supplement the GPS data for interpolation between data points and whenever navigation satellites are not available. In addition, the inertial yaw, pitch and roll information and their rates are used in later calculations to determine the actual direction of each camera""s line of sight as a function of time to accurately determine the direction of objects in the cameras fields of view.
A second embodiment of this invention uses a single FDDI Token Ring LAN network to transmit both the control and data signals. This second embodiment sacrifices some transmission speed in exchange for a single ring network.